Papers with multimodal understanding tasks
Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems require large number of accurate annotations, such as image-level labels and location-level labeling. |
| Approach: | They propose a joint anaphora resolution and object localization dataset targeting visual-linguistic ambiguity. |
| Outcome: | The proposed framework improves visual-linguistic alignment and object localization with one joint model compared to a strong single-task baseline. |
Model Composition for Multimodal Large Language Models (2024.acl-long)
Copied to clipboard
Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu
| Challenge: | Existing methods for creating versatile MLLMs rely on joint training with paired instruction data, which is resource-intensive and challenging to extend to new modalities. |
| Approach: | They propose a new paradigm for multimodal large language models by reusing modality encoders and merging LLM parameters. |
| Outcome: | The proposed model retains the modal understanding capabilities of each original model. |
Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for reducing hallucinations incur a significant increase in latency. |
| Approach: | They propose a task-agnostic attention-guided head suppression strategy that can be seamlessly integrated during inference without incurring significant compute or latency overhead. |
| Outcome: | The proposed approach reduces hallucinations by 2.7x while maintaining F1 and improves throughput by 1.8% compared to existing methods. |
FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on multimodal fashion tasks has been limited by the data in individual benchmarks or has leveraged generic vision-and-language pre-training but have not taken advantage of the characteristics of fashion data. |
| Approach: | They propose a fashion-specific pre-training framework based on weakly-supervised triplets constructed from fashion image-text pairs. |
| Outcome: | The proposed framework is based on weakly-supervised triplets constructed from fashion image-text pairs and is competitive on a diverse set of fashion tasks. |